The carrier changed a rate limit and nobody told us

parcel/ship sits between a storefront and eleven carriers. A customer asks what a two-pound box costs to send to Tucson; parcel/ship asks eleven APIs, in parallel, and answers in under 400 ms or the checkout page moves on without it.

Carriers are not a stable dependency. They change limits, they change error shapes, and they very rarely send an email about it. In June one of them cut its burst allowance by two thirds overnight. The response was an HTTP 400 with a body that read like a validation failure, so for a day and a half the failures were triaged as bad addresses from customers.

Grouping did the work

What broke the tie was the fingerprint. Eleven thousand of those 400s were one issue, not eleven thousand, and the issue's tags said something no log line had: every one of them carried carrier:atlas and every one arrived in the four minutes after a burst.

• One issue, one owner, routed by CODEOWNERS to the integrations team.
• Traces showed the retry storm the limit was actually punishing.
• A release marker made it obvious the change was not ours.

Where it landed

Failed rate requests went from one in ninety to one in 3,400 — the remainder are genuinely bad addresses. Night pages went to zero, which for a team of twenty-two on a follow-the-sun rotation is the number that decides whether people stay.

Tracing every quote costs parcel/ship 38 ms at p95, and they sample at 20% in peak hours. It is, in their words, the cheapest instrumentation they run.